Impact of Outcome Adjudication in Kidney Disease Trials: Observations From the Study of Heart and Renal Protection

Introduction We aimed to assess opportunities for trial streamlining and the scientific impact of adjudication on kidney and cardiovascular (CV) outcomes in chronic kidney disease (CKD). Methods We analyzed the effects of adjudication of approximately 2100 maintenance kidney replacement therapy (KRT) and approximately 1300 major atherosclerotic events (MAEs) recorded in the Study of Heart and Renal Protection (SHARP). We first compared outcome classification before adjudication versus after adjudication, and then reran randomized comparisons using preadjudicated follow-up data. Results For maintenance KRT, adjudication had little impact with only 1% of events being refuted (28/2115). Consequently, randomized comparisons using preadjudication reports found almost identical results (preadjudication: simvastatin/ezetimibe 1038 vs. placebo 1077; rate ratio [RR] 0.95, 95% confidence interval [CI] 0.88–1.04; postadjudicated: 1057 vs. 1084; RR = 0.97, 95% CI 0.89–1.05). For MAEs, about one-quarter of patient reports were refuted (324/1275 [25%]); and reviewing 3538 other potential vascular events and death reports identified only 194 additional MAEs. Nevertheless, randomized analyses using SHARP’s preadjudicated data alone found similar results to analyses based on adjudicated outcomes (preadjudication: 573 vs. 702; RR = 0.80, 95% CI 0.72–0.89; adjudicated: 526 vs. 619; RR = 0.83, 95% CI 0.74–0.94); and also suggested that refuted MAEs were likely to represent atherosclerotic disease (RR for refuted MAEs = 0.80, 95% CI 0.65–1.00). Conclusions These analyses provide 3 key insights. First, they provide a rationale for nephrology trials not to adjudicate maintenance KRT. Second, when an event that mimics an atherosclerotic outcome is not expected to be influenced by the treatment under study (e.g., heart failure), the aim of adjudicating atherosclerotic outcomes should be to remove such events. Lastly, restrictive definitions for the remaining suspected atherosclerotic outcomes may reduce statistical power.

R andomized trials are the cornerstone of evidencebased practice, but scientifically unjustified and costly procedures have led trials to become excessively complex. Clinical outcome adjudication is the process of verification of reported outcomes using source documents (e.g., medical notes) according to prespecified diagnostic criteria by trained clinicians blind to study treatment allocation. It may offer reassurances about the reliability of a trial's findings and have particular value for outcomes with complex presentations with multiple alternative diagnoses (e.g., heart failure), if a precise diagnosis is critical for trial interpretation (e.g., reliably distinguishing between ischemic and hemorrhagic stroke in an antithrombotic trial where treatment effects are expected to differ qualitatively), or if reliability of the reporting source is unknown. 1 However, clinical adjudication of CV outcomes in trials conducted in populations with low prevalence of CKD has been shown to have almost no meaningful effect on trials' observed relative treatment effects. 2,3 In addition to the financial cost of adjudication, which may limit sample size or trial duration, adjudication could also conceivably harm a trial. For example, overly specific adjudication criteria may lead to "true" outcomes being refuted, reducing trial power (where not event driven). Adjudication could also conceivably distract the focus of recruiting sites and trial coordinating centers from procedures which are critical-to-quality, such as maximizing adherence to allocated study treatment and completeness of follow-up. 4,5 As increasing complexity of conducting clinical trials threaten our ability to answer important future clinical questions, 6,7 reviewing the scientific value of expensive time-consuming trial procedures, like adjudication, and fostering appropriate streamlining could help increase the number of practice-changing trials in nephrology and improve current and future patients' care. 5 Recognizing these issues, the Kidney Disease: Improving Global Outcomes guidelines recommend that kidney trials are tailored with "adjudication methods to focus on those events in which adjudication may materially influence interpretation of the results," 5 and the International Society of Nephrology's First International Consensus Meeting on Defining Kidney Failure in Clinical Trials concluded that "the role of event adjudication for kidney failure needs to be evaluated within the specific context of the study planned." 8 Patients with CKD are at increased risk of CV diseases as well as kidney failure, 9 with documentation related to kidney and CV outcomes often collected and adjudicated in nephrology trials. We aimed to use data from SHARP to assess the reliability of KRT and CV clinical outcomes reported by patients with CKD as compared to a gold standard of outcomes derived following adjudicated direct follow-up.

METHODS
SHARP's design and results are reported elsewhere. 10,11 In brief, SHARP assessed the effect of lowering low density lipoprotein-cholesterol with simvastatin 20 mg plus ezetimibe 10 mg once daily versus matching placebo on the incidence of MAEs among 9270 patients with CKD followed-up with a median of 4.9 years. MAEs were defined as nonfatal myocardial infarction or coronary death (referred to as major coronary events), nonhemorrhagic stroke (ischemic and unknown subtypes combined), or arterial revascularization (coronary or noncoronary, excluding dialysis access). End-stage kidney disease was defined as the initiation of maintenance hemodialysis (defined as planned dialysis initiation or dialysis status for $3 months) or receipt of a kidney transplant, and it is referred to as maintenance KRT throughout this manuscript.
The main method of follow-up was clinic-based systematic interviews of participants (or with relatives or clinicians in the event of death) every 6 months. Participants' answers were recorded and coded in real-time directly into clinical terms on a computer-system by trained local research coordinators (usually nurses). Specific descriptive codes for initiation of maintenance hemodialysis and peritoneal dialysis were used, with temporary dialysis distinguished using the code "acute-on-chronic renal failure requiring dialysis." Freetext descriptions, recoded by study clinicians at the central coordinating center, were only used when local research coordinators were uncertain about an appropriate event code term. There was no requirement for direct review of reported events by local physician investigators. All deaths, MAEs, potential MAEs (i.e., events terms which could represent a miscoded MAE, see Supplementary  Table S1 for full details), and KRT events (maintenance or not) were then subject to clinical adjudication. The central coordinating center requested documentation (e.g., hospital discharge summaries, death certificates, clinical investigation reports, vascular access procedures, etc.) for each of these potential outcomes. All mentions of use of lipid-modifying drugs and laboratory measurements of lipids were redacted before documents were made available to nephrologist adjudicators blind to treatment allocation. They followed a standard operating procedure containing prespecified diagnostic criteria. If during the process of adjudicating an event, the adjudicator came across information about a potential clinical outcome that had not been reported, they would create a new report in the study database. Adjudication documents were available in >90% of potential outcomes.

Statistical Methods
In this report, tabulations of preadjudication versus postadjudication outcome categories (based on the first relevant event in each category) were constructed and the proportion of preadjudicated outcomes refuted by adjudication was calculated. Agreement statistics were not presented because the 2 outcome datasets were not independent. 12 Differences between event dates before and after adjudication were presented as categories, including the following: exact match, as well as 1 to 7, 8 to 30, 31 to 90, 91 to 180, and >180 days. Randomized comparisons of the effect of allocation to study CLINICAL RESEARCH intervention using intention-to-treat time-to-first event log-rank methods were rerun, presenting rate ratios (RRs) with 95% CI. 10,11,13,14 These analyses were conducted for maintenance KRT and MAE outcomes based on preadjudication participant reports and compared with the published postadjudication outcome data. 10,11 The 95% CI for the difference between RRs was estimated using bootstrap methods. For MAEs, analyses were performed using refuted, unrefuted, and "identified by adjudication" events only; and considering separately patients not on dialysis or on dialysis at randomization. All analyses were conducted using SAS version 9.4 (SAS Institute Inc) and R version 4.2.2 (R Project for Statistical Computing).

Maintenance KRT
Among the 6245 patients not on dialysis at randomization, 2115 maintenance KRT events were reported, of which less than 1% (n ¼ 28) were refuted by adjudication (Table 1). Only 1 of the 504 reported kidney transplant events was refuted. Adjudicating 312 events reported as acute-on-chronic renal failure events requiring dialysis identified an additional 54 maintenance KRT events. Differences in reported dates of start of maintenance KRT between datasets of >30 days were uncommon (<7%, Supplementary Table S2).
Randomized analyses using preadjudication data showed that allocation to simvastatin/ezetimibe had no effect on risk of maintenance KRT (simvastatin/ezetimibe 1038 vs. placebo 1077; RR 0.95, 95% CI 0.88-1.04), almost identical to results from adjudicated direct follow-up (1057 vs. 1084; RR ¼ 0.97, 95% CI 0.89-1.05; Figure 1). The 95% CI for the À0.01 difference between these RRs was À0.03 to 0.00. Similarly, RRs estimated from preadjudication and postadjudication events were almost identical for the outcomes of initiation of maintenance dialysis and receipt of a kidney transplant considered separately.

MAEs
Among the full 9270-participant SHARP cohort including patients on dialysis, 1275 first MAEs were reported, of which 25% (324/1275) were refuted by adjudication (  Table S4). In analyses comparing Figure 1. Effect of allocation to simvastatin/ezetimibe versus placebo on maintenance kidney replacement therapy and major atherosclerotic events using SHARP adjudicated and preadjudicated follow-up. Log-rank methods were used to calculate the rate ratio and 95% confidence intervals. *Excluding those on maintenance dialysis at baseline. Risk ratio for the major atherosclerotic events refuted by adjudication was 0.80 (95% CI 0.65-1.00). CI, confidence interval; CHD, coronary heart disease.

CLINICAL RESEARCH
WG Herrington et al.: SHARP Adjudication patients by baseline dialysis status, the heterogeneity test P-values for the effect of simvastatin/ezetimibe versus placebo on MAE was 0.24 for adjudicated MAE, and 0.74 for preadjudicated MAE (Figure 2).

DISCUSSION
These analyses from the double-blind placebo-controlled SHARP trial found that information about events reported by participants to trained research coordinators, without review by local or central clinicians, provided consistent estimates of relative treatment effects for the outcomes of maintenance KRT and MAE. In particular, findings from preadjudication and postadjudication outcome data were almost identical for kidney transplantation and very similar for initiation of maintenance dialysis and revascularization procedures. They provide a particularly strong rationale for refining consensus conclusions regarding adjudication of kidney failure to support more streamlined approaches. 8 These would ideally encourage not adjudicating KRT outcomes from adequately blinded trials in CKD populations, because most events are correctly categorized when reported.
The presented SHARP observations based on populations from North America, Europe, and Asia provide the largest and most generalizable experience of adjudicating KRT and CV outcomes in a CKD trial. The SHARP CV data also confirm that previous outcome adjudication experiences from diabetes and cardiology trials are relevant to CKD trials. SHARP results mirror those from the 15,480 participant A Study of Cardiovascular Events iN Diabetes (ASCEND) trial in diabetes 15 and from 42 cardiology trials (including over 200,000 participants) which together have consistently reported that relative treatment effects on CV outcomes were not materially Figure 2. Effect of allocation to simvastatin/ezetimibe versus placebo on major atherosclerotic events using SHARP adjudicated and preadjudicated follow-up, by dialysis status. Log-rank methods were used to calculate the rate ratio and 95% confidence intervals. *Heterogeneity between patients on dialysis and not on dialysis: c 2 1 ¼ 1.35 (P ¼ 0.24) for adjudicated follow-up and c 2 changed by clinical adjudication. 2,3 Future extension of the presented research using a similar approach to our recent analyses from the ASCEND trial 15 is now indicated to assess if unadjudicated routinely collected hospital admission data from different regions of the world are sufficiently reliable to be used as the sole method of follow-up for KRT and CV outcomes in CKD trials. SHARP data may provide a rationale for not requiring adjudication of MAEs. Despite substantial recategorization during adjudication, this had no material impact on the RRs for confirmed, refuted, and newly identified outcomes, which were similar. In fact, it is possible that our approach to CV outcome adjudication may have reduced SHARP's power to detect treatment effects, especially in subgroups. The similarly-sized RRs for the 1145 confirmed MAEs and the 324 refuted MAEs suggest that many of the CV events that did not fulfill the prespecified criteria for confirming MAEs, most of which led to hospitalization for several days, were prevented by lowering low density lipoprotein-cholesterol and therefore represented modifiable atherosclerotic disease. It was notable that agreement between preadjudicated and postadjudicated datasets was less strong for major coronary events than other MAE components (nonhemorrhagic stroke and revascularization procedures). This may reflect the challenges of distinguishing between atherosclerotic and nonatherosclerotic cardiac events based on clinical findings in patients with advanced CKD. 16,17 The corollary of these SHARP observations is that if CV outcome adjudication is deemed scientifically important in a CKD trial, then outcome definitions for major coronary events should allow for inclusion of atypical presentations and incomplete information. Avoiding overly strict and overly specific adjudication criteria could reduce the chances of "real" outcomes being refuted and maintain trial power. This may have been particularly true among patients on dialysis whose reported MAEs were more likely to be refuted. Both the relative and absolute effects of lowering low density lipoprotein-cholesterol on MAEs in patients on dialysis may have been underestimated in SHARP ( Figure 2).
In conclusion, these findings from SHARP lead to 3 key conclusions with practical implications for outcome adjudication in trials in CKD. First the data provide a rationale to stop adjudication of maintenance KRT in CKD progression trials because the vast majority were correctly classified on reporting. Second, when an event that mimics an atherosclerotic outcome is not expected to be influenced by the treatment under study (e.g., heart failure), the aim of adjudicating atherosclerotic outcomes should be to remove such events. Lastly, restrictive definitions for the remaining suspected atherosclerotic outcomes may reduce statistical power.

List of the SHARP Collaborative Group
Members of the SHARP Collaborative Group are listed in the Appendix of Baigent et al. 10 DISCLOSURE CTSU has a staff policy of not accepting honoraria or other payments from the pharmaceutical industry, except for the reimbursement of costs to participate in scientific meetings (www.ctsu.ox.ac.uk). All authors report a grant to their institution from Merck & Co., Inc., Whitehouse Station, NJ USA to conduct the SHARP trial. The funders had no role in the conceiving or authoring of this paper, nor the decision to submit.
WGH & CH wrote the first draft of the manuscript. All authors interpreted the data and commented on the manuscript.

SUPPLEMENTARY MATERIAL
Supplementary File (PDF )  Table S1. Cardiovascular events selected for adjudication to identify potential missed major atherosclerotic events in SHARP .  Table  S2.
Comparison of preadjudication and postadjudication follow-up for deaths in SHARP. Table S4. Fact of hospitalization, duration of hospital stay, and effect of allocation to simvastatin/ezetimibe versus placebo on refuted, unrefuted and "identified by adjudication" major atherosclerotic events. Table S5. Tabulation of major atherosclerotic events before and after adjudication in SHARP, by dialysis status.